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Summary of Montana STEP Project Approach to Course Reform Evaluation 

Introduction 

The Montana Systemic Teacher Excellence Preparation (STEP) Project is a National 
Science Foundation (NSF) Collaborative for Teacher Preparation (CETP), funded in 1993, as an 
initiative to accomplish systemic change in mathematics and science teacher preparation in 
Montana. The STEP project involves faculty from the Montana University System and tribal 
colleges; the state Office of Public Instruction; K-12 teachers; the Montana Science Teachers 
Association (MSTA); and the Montana Council of Teachers of Mathematics (MCTM). The goal of 
the Montana collaborative was to bring about large-scale improvement in the preparation of K- 12 
science and mathematics teachers in Montana and to serve as a national model for rural areas with 
significant minority populations. To this end, the collaborative uses a team approach to redesign 
content and methods courses in mathematics and the sciences for preservice teachers; creates model 
teacher preparation field sites in K-12 schools; recruits and supports future Native American 
mathematics and science teachers; uses Montana’s extensive telecommunications network as a key 
component in the development of teacher preparation models for rural schools; and provides an 
early career support (mentoring) program for mathematics and science teachers in rural settings 
during their first four years of service. 

A team approach was used to redesign mathematics and science content courses, 
and methods courses for pre-service teachers. Teams, which include faculty, graduate 
teaching assistants and K-12 teachers, meet regularly to redesign or create undergraduate 
courses at five Montana University System campuses. Course reforms are designed to 
align with research supported “best practices” described in STEP’S “Table One: Reform 
Goals for Mathematics and Science Teaching.” This table was constructed collaboratively 
by a state-wide faculty group, 1992-93. This table, from the original STEP proposal, is 
the guiding framework for mathematics and science teaching reform. Appendix A contains 
STEP’S “Table One”. A total of fifty-two (52) courses were revised or created between 
1993 and 1996. Note: Descriptions of individual course initiatives, arranged by campus, 
are found in a Course Revision Catalogue available from the STEP project. 

The project was challenged to document progress made in revising the 
undergraduate courses for preservice teachers. Information was required to inform a 
variety of people including university administrators, content departments, faculty, project 
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Pi’s, campus coordinators, STEP’S National Visiting Committee, the NSF and interested 
others. From the very start, interest in the assessment of revised courses exceeded interest 
in assessment of other project components (e.g. the K -12 model school sites or an early 
career teacher mentoring program). 

Approach to Evaluation 

The assessment plan was an embedded and integral part of the course reform 
process based on a belief that assessment and evaluation had the potential to make a 
positive difference. According to Students at the Center (Musil, 1992), fifty-two percent of 
the nation’s colleges and universities reported that assessment led to changes in curricula or 
programs. Faculty members involved in assessment reported that their view of teaching 
and their activities in the classroom had been affected by involvement in assessment. We 
embraced a view of assessment that tied it firmly to learning and offered genuine hope for 
real undergraduate reform. A rich and varied set of assessment methods was planned that 
would coalesce to: improve rather than prove; view student experience over time; 
incorporate multiple methods and sources of information; focus careful attention on how 
information was used; provide occasions to talk about and interpret information; involve 
faculty members; and involve and listen to students. 

The evaluator practiced a participatory approach to assessment and evaluation that 
involved project leaders and participants in identifying the kinds of information needed to 
guide the evolution of the program and ensure its success (Patton, 1986; Guba & Lincoln, 
1989; Stevens, Lawrenz & Sharp, 1993; Scriven, 1993). She worked with the staff to 
identify key formative evaluation questions, and provided a continuous stream of formative 
data to encourage improvement. 

Sample questions that emerged included: Is an expanded group of faculty recruited 
to the reform process? What faculty beliefs about learning are changed? Do new faculty 
members and graduate teaching assistants find the course reform team approach helpful as a 
network to learn about STEP project goals? Are faculty designing and carrying out 
classroom and program assessments themselves? Is dialogue and discourse helping to 
define campus action and research agendas for teaching and learning? Is the formal reward 
system changing in response to a shift in campus orientation to support quality teaching? 
What is the nature of communication among the faculty, departments, and colleges about 
course reform ? 
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Inclusion & Empowerment 

Assessment was intended to be a central and cohesive element in the reform 
process. Faculty members have long practice in making judgments about student work; 
their expertise in doing so is crucial in deciding what questions assessment should focus 
on, what the collected data add up to, and what should be done to improve instruction. 
Since the single most important route to improvement is through the classroom, faculty 
members in particular were active participants in the assessment process. The assessment 
was not envisioned as primarily an administrative task, but rather an educational process. 
The assessment moved from the “inside out” rather than from the “outside in.” People 
knowledgeable about pedagogy, methodology and who were active in the design and 
development of the project played key roles in the assessment process - project directors, 
campus coordinators, and faculty members. 

The fifteen STEP campus coordinators (three faculty from each of five Montana 
State University campuses) provided early feedback to the evaluator on the assessment plan 
and instruments to ensure that they reflected the goals of STEP’S “Table One” Framework 
for Mathematics and Science Reform (see Appendix A). This table, from the original 
STEP proposal, served as the guiding criteria for constructing assessments and doing the 
evaluation. Assessments were done in each of the 52 STEP courses. All faculty teams, as 
well as their students, from the departments of education, mathematics, and the sciences at 
each of the five Montana University System campuses participated in assessments. Teams 
of faculty were invited to review and comment on evaluation data and draft reports. 
Appropriate data and reports were provided to faculty teams and campus coordinators. 

STEP Course Revision Catalogue 

As the number of courses engaged in the reform process grew, it became 
increasingly difficult to keep campus coordinators and others informed. In response, a 
STEP Course Revision Catalogue was created with basic information for each course. The 
catalogue, arranged by campus, lists the title, rubric/number, education program (for 
elementary and/or secondary preservice teachers), date revision process began, a brief 
description of the goals for the reform, and the name of the faculty team leader. This 
catalogue was helpful to those people interested in knowing what courses were part of the 
STEP Project and the nature of the reforms. 
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Collecting Data 

Cooperation and input from the course reform faculty at each of the Montana 
University System campuses was critical. Although it required a significant amount of 
travel, getting acquainted with faculty and developing a comfortable and productive 
working relationship was an essential activity in the first project years. Once that had 
happened, data collection for subsequent years was considerably easier. Communication 
by telephone and e-mail worked well once a personal relationship existed. 

Multiple Data Sources 

Using multiple data sources was recognized as important (Worthen & Sanders, 
1987; Fetterman, 1988; Frechtling & Sharp (Eds.) 1997). Methods included: (1) semester- 
by-semester faculty team surveys; (2) visits to reform team meetings; (3) faculty interviews; 
(4) student interviews; (5) class observations; (6) a survey of all students enrolled in 
reformed courses Spring Semester 1996, and (7) document review of course syllabi and 
other instructional materials. Assessments involved course reform faculty teams, as well as 
students, from the departments of education, mathematics, and the sciences at each of the 
five Montana University System campuses. Faculty were engaged in instrument design; 
semester-by-semester surveys; interviews; and class observations. Students regularly 
contributed to focus group interviews and provided survey and questionnaire responses. 
The resulting data were analyzed with criteria established in STEP’S “Table One” reform 
goals for science and mathematics education. Note: Reports and assessment instruments 
are available from the author. 



Impact 

One result of evaluation efforts was a significant increase in university faculty and 
K-12 teacher participation in course reform evaluation. For example, Montana university 
system faculty visited each others’ campuses and attended reform classes as peer 
observers. The observations were made within and between the departments of education, 
mathematics, and the sciences. Also, area K-12 teachers observed revised classes and 
shared their findings with university course reform faculty teams. In addition, campus 
coordinators from different university sites visited revised classes at their home campuses 
and at other Montana campuses. 
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The STEP project steering committee, with representatives present from each of the 
university campuses, endorsed the campus peer visit initiative. Although three statewide 
course reform conferences (1993, 1994 & 1995) had provided important and meaningful 
opportunities for discussions and interactions among the faculty from the five STEP 
campuses, the steering committee concluded that on-site visits, where visiting teams would 
immerse themselves in the reformed learning environment of another campus by attending 
classes and engaging in focused discussions with their colleagues, would enhance 
programmatic collaboration of the STEP Project. 

Faculty groups interested in learning more about a STEP reformed class or classes 
at another Montana campus were invited to participate in cross-campus visits. While visits 
could be arranged for an individual, two or more visitors traveling together from one 
institution was suggested. New faculty members or faculty members who had not been 
involved with STEP at each campus were especially encouraged to participate. 

Campus visits were arranged for peer faculty with similar course/content interests 
from different campuses to (1) attend and observe STEP reform courses and (2) participate 
in focused discussions with STEP reform faculty teams. Teams generally included 
instructors, graduate students, and K-12 teachers who had worked on one or more course 
reform effort. Visitors often had the opportunity to attend other STEP campus group 
meetings or events. 

Much was gained by sharing individual faculty team efforts. Often faculty 
members on separate campuses were working on parallel problems or initiatives. Creating 
a structure for observations and dialogue between colleagues promoted STEP efforts at 
each campus. Expanded class observations at all sites: (1) involved more faculty and 
administrators in documentation of reforms; (2) increased dissemination of reform 
initiatives; (3) created an information base to educate peers about reforms; (4) supported 
faculty interactions between campuses (especially for teams working on similar class 
revisions); and (5) recruited additional faculty at each site. This initiative expanded faculty 
involvement, interaction and collaboration across the state. 
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A Judicial Model 

In a judicial model, the majority of evidence favors one answer. While there may 
not be a definitive answer, the gathered information from different sources seems settle on 
one side. In STEP’S evaluation of course reform, data from revised courses was analyzed 
against “STEP’S Table One” criteria. Information from the surveys, interviews and 
observations provided cross-referenced evidence on the frequency that “Table One” 
strategies are used in revised classes. Because data provided abundant evidence that STEP 
strategies from the “Table One” framework are implemented in the revised courses, we 
came to call this a “judicial” model of evaluation. 

The name “judicial” seems appropriate since a preponderance of evidence was 
gathered and weighed to create a well documented account of the Montana STEP Project’s 
reform process and accomplishments. A strong case was built to support a belief that 
reformed-style instruction now occurs in STEP revised courses. 
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Appendix A 

“Table One” 

STEP’S Reform Goals for Mathematics and Science Teaching 

STEP teachers at the colleges, universities and K-12 schools will: 

1. Use curricula that show relationships between different fields of 
science or mathematics, and between science, mathematics and other 
subject areas, such as social studies or reading/language arts. 

2. Actively engage students in inquiry, problem-solving, and model 
building. 

3. Use specific strategies found to be effective in engaging female and 
minority students, especially Native Americans, in mathematics and 
science. 

4. Recognize the progression in learning from concrete to abstract, 
providing ample opportunities to work with manipulatives and 
hands-on materials, and first-hand experiences in laboratory or field 
settings. 

5. Provide regular opportunities for social interaction and group work. 

6. Use strategies to identify and build upon students’ preexisting ideas. 

7. Introduce real world applications of mathematics and science. 

8. Use appropriate technologies, including graphing calculators, a 
variety of computer applications (e.g. modeling, laboratory interface 
systems), telecommunications networks (e.g. Internet), and video 
technologies. 

9. Use assessment techniques that are challenging, varied and allow 
students to make some choices about how best to demonstrate their 
learning. 

10. Demonstrate effective management in an active classroom. 

1 1 . Promote a culture in which teachers' efforts to promote students' 
involvement in mathematics and science are actively supported by 
school administrators, parents and other members. 
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